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ABSTRACT ' • ' 

^ ' The National Assessment of Educational Progress 
studies selected research efforts in the agriculture, econoMdcs, and 
health fields to discover ^successful strategies for the dissemination 
and adoption of educational research. Interviews vere conducted Mith 
people familiar With the Agricultural Extension Hcvement, Health 
Examimation Interview Su^rveys, and the Consumer Expenditure Survey^ asl^ 
veil as with people experienced in the dissemination ctf educational 
research information, (five appendixes contain lists of participants 
in the study dnd descriptions of the study methods and resources.) 
Findings and recommendations included the foj^lowing: (1) Careful 
planning of research directions to avoid duplicate efforts requires a 
good communication structure. (2) Thorough planning and development 
of methodology and instruments fdt data collection includes site 
visits^ pretests, and rigorous standards. (3) (positive staff morale 
needs to be developed; a variety of effective jieasures are- available. 
(4) Efficient dissemination of research findings involves the use of 
computer information ai^d retrieval systems, reports, publications, 
etc.; a model fyst.em vas found to be the O.S« Department of 
Agriculture's Cur^rent Research Information System. (?) The effective 
communication of research results to' key decisionmakers demands the 
use of easily comprehensible language and format and the proximity of 
the researchers to the policy makers to facilitate re^lar contact 
and appearances at hearings. The success of adoption of research 
inforlmation depends on linking, agents which assume incireasing 
importance for education since so many innovations are at the local 
level « (Specifd? strategies found to be effective are listed and 
discussed un^er each of the five categories /cf findings.) (ELG) 
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OVERVIEW 



/National Assessment of Educational Progress (NAEP)*, Denver, Colorado, 
conducted a study from A^il 1-Augijst 31, 1977, of, selected research 'efforts • 
in the fields of agriculture^ economics, health and education. 

The study was dojue with the\hope of discovering strategics that have been 

' ■ • ' 

successful in the .dissemination and^adoption of reseai^^h information. 

Interviews' were conducted with people familiar with the Agriculture Ex- 
tension Mpyement, Health Examination Int^rviev Surveys and the Consumer Ex- . 
penditure 'Survey. Interviews were also conducted with people wiio have experi- 
ence with disseminatio^ of educational researciNiriformaCion. 

In or^der to reach a variefiy of audiences, the^study wa^^i^nritten in 



journalistic style with an attempt to udnimi^e the usVof technical terminology. 
The study was not designed to be a technical document bi^t^ rather a general 
description of research techniques effectively used .in agriculture^ )ytalth and," 



economics fields! 



\ 



} ■ 



Background information for the -report was> collected by a sttidy group in-^ ^ 
^ / ' » . ' ^ 

eluding: Gloria Frazier,, applications specialist; Dick Hulsart% area deVe^op^ 

ment coordinator; and Sara tSiocran, research, assistant. Based on iJifonnaflcin 

provided by the study group, Linda Stahl, journalist, wrote the repor't,- Th^ 

study was financed by a grant from the Ford Foundation. % \ . • 

J * . ^ ''4- c ,A 



" J \^ 

*NAEP,is funded by and under contract with the^National Pentfer for Education * 
Statistics (NCES), U,S. Office of Health, Education and Welfare,^ Washington,/, 
D.C. It is a project of the Education Commission ott the STtateS .fECS) , "Denver,- ' 
Colorado. This study does not necessarily represent the views of NCES (Jr.'EtJS^. 



SUMMARY OF TH? FINDINGS . 



Key findings . that emerged from the sttxdy, and the/major strategies f6t 



dealing effectively with the findings, ^are piitlined below. Further elaboration 
of the findings are included in one, or more, chapters within the report. 



Careful planning of directions for research includes: 

Development of an overall research planning system. 

Development of special studies to identify heeds anci investigate new 
approaches. 

* * 
-^Thorough examination of past and current research. 

/ 

Review of.exjLsting and planned-for/ federal and state projects/programs. 
Suajveys of impact of current research on users and .attitudes of users , 



about future research. 



Reyiew of requests for information about research. 

Constiltation with various technical and administrative experts from, 
outside organizations as well as within the systeia. 

Participation in national, regional and local conferences where reseajrchers 
can share ideas with potential users of the data, as well as with other 
researchers. 

Contact with private ^organizations and other goveimment agencies conducting 
similar or related reseatch. 

Communication with federal and state legislators through persona^ contact 
and Attendance at "hearings. 

Examination of legislative records and reports. 



iy 



Thorough planning and development of methodology for data collection Includes: 

Involvement of experts with a wide range of experiences in development 
^and review of instruments to be used*in data collection. 

• Examination of d^ta collection methodology of other research organizations. 

Extensive pilot testing of instruments and data co.llection techniques.^ 

Revision of instruments and data collection techniques based on oilot 
testing. ' . - ' • , ^ 

Thorough "training of personnel involved in collection of research'^ata. 

r 

Development of positive ^^taff morale includes; ^ \ ^ 

Providing staff* with opportunity for; a^ariety of job positions and exper-* 
ienc^s." * . , 

' Budgeting time, and funds to develop research methodology. ' • 

• ^ > 

Inforaingy^^^aff of how the results of the research are utilized. 

Ei^couraging staff contact with other researchers both within at^d externa 
^ to organization." ' 

Providing opportunity for co-authorship of articles with t)ther professionals .j 

Jeveloplag ways to keep staff informed about each 'others' work' withiii the 
. organization* ^ ■ . . \ 

^ laereasing knowledge about processes neciss^ry to insure utility ot research. 

• Considering starff contributions as important as that of outside cion^ultants. 

Efficient overall* dissemination* of research findings includes: 

* .Involvement of potential users ,in planning, analysis and reporting df^ 
research. - ' ' ' ^ , 

Development of an ^easJ Ay accessible computer ijiformation retrieval system. 

• ■''■'■'/ Y ■ 

Publication of reports that emphasize. key findings and important trends in 
. language appropifiate for target aud^nce. . » "-^ . 

^' Use 'of a^ variety of communication devices — s^raff-'publicatidQs, popular 
magaz-ines 4ind newspapers., prQfessio,naJ journals, T.V..,and radio. 
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Cooperative dissemination' activities with other researchers in private 
aad public organizations. . , 



Regular coiftact with med\a people. / 



Providixig testimonyC^ef ore 'legislators. 

I^rovidin?\policy-relevant data for federal, state and local advisory 
councils and task forces. 

Mictofiching, microfilming and taping of techni*cal data. • 

avelopment*5df agents dr organizations that link researchers with users 
5f research restilts. ' • ^ 



Effective communica tion of research results to key decision makers includes: 

Maintenance of •continuous personal contact with key decision makers 
thorough seminars,,^ telephone contact,^ individual meetings and attendance 
at hearing^. > ' r 

Assignment of personnel within res^earch bfgknizations to determjLne needs 
or decision makers and to provide Research results in format that decision 
makers can use. . ^ • ♦ 

Development of reports of research results that suiflmarize significant 
^ fixxdings, outline alternative solutions and/or directions, and provide ^ ' 
information, o^n other studies that have supporting or conflicting findings. 

Release of selected research results to 'key decisioji 'makers -before • ^ 

releasing to the media and public: 

«' 

'Making clear ^to key deAlsio'tvniakers the importance of proposed or 
conducted research. ■ . 

Development of a journal that describes implications' of major- research 
studies in a particular field. 

Establishment of panel of rioted researchers to-judge the validity of the 
-findings of major research studies^ ^ ' \ ' . ' 




CHAPTER 1 

THE PROBLEMS OF EARLY ADOLESCENCE , • 

In the United States, educational research and development is at an 
wkward stage — .gangling, uncoordinated dnd often appearing uncertain in both 

o 

• ^ \ 

;\purpose and direction* 

- / • \ ,, - ■ v ■ • • 

No wonder. It has existed as an organized efat'erprise^ in this country for 
only twenty years. Compared with American research efforts in other fields, 

such as medicine and agriculture, it is in early adolescence. \ 

* \ ' ^ i ' 

It is -not only a fledgling system facing the normal problems of growing up 

< • 

but it is a system that suffers from an image , problem, too. It might be de-* 

** ^ * , * ' * 

.scribed as the image of a child who did ijot live up "tO .the unrealistic expecta- 
tions of ambitious parents. * / ' < 

"Educational research aqd daveloj)ment' will continue to be regarded with 
a dim view for vrfiat il^annot do. The sixties^oversold it jmd'now we are get- 
tin^*" the reaction...,'* said' Corrine Rieder, an associate directot^at -the 
National Institute of Education (NIE) . 

She and others in her field expreiised that view at a Washington conierence 
held in the spring of 1^77 by the Inst^.tute far Educational Leadership in co- 
. operation with the American Educational Research Association. 

X I^.th® sdLxties, when the educational research field grew .most rapidJjy, 
there^were several major souices of impetus. The 1963 • Vocational Education 
Act ir^cluded substantial money for research. ' The Elementary and Secondary 
Education Act, passed in 1965, and its subsequent amendments, provided^ funds 



' for the research, development and deaoMtration o'f various approaches to sdrlv- 



Ing/the -educational problems of disadvantaged and impoverished students. 

In a two-year period, 1965 to 1967, twenty-one research and dfevelopme^t. 
centers, twenty regional educational laboratories and more than one hundred 
research training i^ograns blossomed. Educational Resources Information 
Center (5RIC), ^the national information storage and retrievfil system of*ed- 
ucation, hummed into operation. • . r\ 



'0 



Finally, in 1972, the- federal government took the leadership in funding 
educational research by cheating NIE. Hopes ran high.' 

However, the NIE budget plimged from $110 million in' 1973 to /$ 70 million 
in 1976. The great hopes for continuing and enthusiastic support for the ^ 
educational research field began to wane. Today the NIE budget, $89.2 million, ^ 
still below the original funding level, is awaiting congressional approval. 

* The National Council on Educational Research points out that only on^ 
half of one percent of the total national expenditure for education is spent 
on research. ,In the agricultural field,* it is five percent. ' ^ 

Part of the problem has been that some observers, anticipated "instant" 
results froth educational research — a-miractilous panacea for the Schools' 
problems In two or three years. They did not stop to think that in other 
fields major breakthroughs c^e only after many, many years of research and ^ 
development. " , " • , 

Besides this impatience on the part of various groups, "another problem 
has plagued educational re^arch. Some school systems of the nation seemed to 
lack incentive for adoption of new techniques resulting from research efforts. 



A 



U.S. Senator Claiborne Pell, chairman of the Senate subcommittee*' on 
Education, Arts ^nd^^Hu^nities , explains it: / • 
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Under 'the National Institute bf Health, « if a new life-^ 
- saying device ot. medical treatment is discovered, there is" a strong 
move' to adopt it in every hospital throughout the country, for^it 
sav€9 lives. Shojild research and development in the space or 
^ ' defense field create 'a new technique, there Is a very Itrong 

reison to -adopt it because of the cost-savitfg factor and the profit 
motivation." v ' . - 

'*Such is not the case with education research and development.- 
It is not a simple matter -for a local educational agency ,to adopt 
^ a new method of lixstruction or a new type of teaching machine*. 
Machines cost* money. Teachers have to be retrained. This, too, ^ 
^ costs jDoney. Yet the corollary with industry and cost savings and 
^ profits does hot apply ^ Education budgets ar,e tight end are not . 
prone to. swift .adjustments, • In /other words, there is a clear- 
cut lack of incentive to adopt that wjiich- has been created,"/ 

. There also have been serious problems with the ability of educational re- 

searchers to express their findings in terms readily understandable and usable 

« 

to educational practitioners or key decision makers molding the course of ed-* 
ucation activities in the nation, ' " ' ^ , 

In a recent interview, ^Chester Finn, legislative assistant to U,S. Senatoi 
Daniel Patrick^ ^Joynihan, "cited the missing ingredients in, the utilization of 
educational research data: the absence of shorty concise 'reports iuNsimple 
language; the frequent failure to supply ^ key data supported by evideilce; and 
the almost totaT failure of educational researchers to frfcjid oa"^e 'policy^ 
implications of th^ir work, ' * ' * 

George Kaplan oj^ the Institute for Educational Lea<iership, Washington, 

D,C., -said educational research reports are too lengthy, technicjU. and steeped 

in jargon and focus more on individual * facts than process to a fauJj.t. 

The^e men, as well as others interested in the problems of educational 
. • , _ • 

research , ja.ffectiveness, said that researchers should suggest possibla uses for 

.their da'fia.and provide alternative avenues to seeking solutions for educational 

- » . ' . A 

problems. ^ ; • , 



-\ • • • ■ ' - ^ 

Pina sa^id the educational' research system might be aided in several ways.. 
Among shis chief suggestions was the develoj^p^ of a Journal, dedicated' to 
educationil research, that, describes Amplications of major research 'studies in 
language understandable, to key decision makdrs and that proyides for possible 
alternatives ani/or directions' to solving problems. 

He suggested that educational researchers in reporting their data to - 
legislators need to not only indicate posslBle Implications of the k^a but 
also, describe ^ther studies which either support or contradict 'their results. 

Finn strongly supports the development' of a panel or academy of recognized 
experts who would have as their, task the examination and review of major 
r^earch results with-the ultimate goal of providing assurance of the validity 
of the' data, i . • . . ' 

U,S,/^epreseht;ativi John B^rademus, House majority whip and member of the' 
House subcomnoittee on. Elementary, Secondary and Vocational Education, has ' 
wirittep' about another^factpr that hasr weakened ^educational research efforts. 
He cites the f jr'equent failure of' professional groups in the field of education, ^ 

icluding those o£ teachers, chief state schooll officers and even researchers 
th^elves to speak out strongly for educational reseairch as a critic/l area 
of activity in the education field, . ' " ' 

The group that did this study en,<jountered the' phenompon of educatibrial ' 
researchers^who are not being sufficiently supported b^ pther .educational groups* 
or^ for thdt matter,' by each other, The study group also detected the existence 
of a weak, professional self-image among some educational researchers. 

An informal survey of educational researchers at the National Assessment 
of Educational Progress (NAEP), Denver, .Colorado., rlve^led some interesting . 
indications of how educational researchers might be viewing themselves in 




the. context of the wider research ^conaounity in this nation.' 

On one siirvey item,- nine teeji participants, were ask^d: ''Jn terms of^recogr 
nition of the importance of research within the profession, 4o you feel that 
educational^ researchers receive l^s, about the same or more recognitioji than 
the foflowiag types of researchers'?" The list included agricultural,, a^ro- 
space, biologi^at, economic, industrial, medical and sociological researchers. 

A jaajority of the educational research prof eTssionals responding to the 
survey indicated that within their^profession they receiv^^I^SS recpgnition 
for their work than, do researchers iln all of che othet fields listed except 
sociology. ' ^ y s ' " 

The survey participants also listed conditions or problems that they felt 
migSt impede the use of educational research^ data by legislators, local school 
district persotinel and state departments of education* The ma^jor reasons 
cited were a lack of funding to do adequate, researclj, a failure t(y interpret 
results^ a failure to put res(earch results in a fo,rm that can be used, a con- 



flictv in results from varipus^^Hjlpardh. projects^ a lack o£^ researchv experience 
by potential users, ^a lack of ^» to^do^adequai^e^ research and a lack- of 
people to ca?ry through on th^f results of research* ^ ^ ^ 



. During the -course of this study, one of the NAEP rles^caRers revealed ^ ' * 
that she had carefully reviewed thk educational cesearc^ knowledge ctf thirty 
""^eachers^ at seven public schopls in* a major city. She wanted to. know what 
research t;hey were aware of and what research findings they were putting to 
use ia their teaching'.' ^ ^ ' ^ ^ 

Only one of nhe thirty elementary and secondary teachers intervieOeSK • 
read, an fducatio^al research publication regulaply; five out* of the thirty tfid 
so* periodically; and six, who happened to be men*^rs of educational. professional 



iaasociations, read.— on thia average ~ only one research article, or book on 
the average per jear. ^ - 

. Twenty-nine of th6 thirty did not know the name of any educational re- 

■ " " ' . • ;. , ■ " ■ ^ ■■ * , 

/ sfearcher. ' Almost ail of the teachers were doncemed about what -researchers 
^ had- to^ say about f the relationship between class size and teacher effective- 
^ ness, but none knew of any research on the topic jj^ ' , . , 

This has by no means been an in-depth .review of the problems facing the * 
fledgling educational research community. The study highlights. soMe of .the 

* » 

^' major prob^lems. This is a summarization of evidence that indicated the ado- 
lesaent 'educational research enterprise is troubled and* searching for direction 
and maturity. ' . . ' ^ .. ' • 



CHAPTER 2 
APPROACH TO mSSION . 

;n its .third annual report, the National .boiincfl on^ Educational Research, 
which provides policy guidance to^ NIE, .tioted some of the fair -and unfair com- 
parisons that have been made with educational research and' other research ' 
efforts: 

f the report says in part: . 

"Because science- has provided answers in such diffuse fields as* 
f space exploration, military hardware,* agriculture production and 

medicine, people ^of ten assume that science also can provide answers 
^ for education. -Sometimes it canw But systematic efforts to' build 
■ an educational research enterprise are barely twenty years- old. 
The accomplishments of space, medical, agricultui-ad ind milijtary 
science result from long and concentrated periods of research 
system building. And all of these" sjrsttenis have produced their . . 

share of failures and unfulfilled hop€s^'' 

The report goes on to say that major research systems, such as agriculture, 
have certain important features that the* educational research system still is 
lacking,. These features are a "s9lid core of high quality" researchers and 
developers, a "solid base" of scientific knowledge and technology and a systie^ 
^^for linking research and practice* \ * 

The NAEP study chose to examine selected research efforts 'in health, 
agriculture and ' economics . " These selections were based on several factors, but 
one- very important factor was that each of these systems shows avidenc'e of using 
research data to influence policy decisions ~ a skill t;he group found sorely 
lacking in the educational research community. Thus, paramount in the group's 
examination was an effort to learn how" these diverse disciplines succeed. in 



Influencing decision makers, as well as practitioner;s, in the field. 
^ Selections were-TTLso based on evidence that the research data were used 
-Nationwide and translated into practice in the field. ' ^ ■ 



In the field of economics, the group studied the Consumer Price 'index, (CPI), 
aljonthly statlitical measure of the average change in prices of goods and ser- 
vices purchase^ by urban wage earners and clerical workers for day-to-day 
living. The examination of the CPI ^ed to a look at the Consumer 'Expenditure 
Survey (CES) . Bo'th CES and CPI are conducted by the' U.S. Bureau "of Labor Sta- - 
tistics. 'CfeS is a survey of consumer expenditures, savings and income that 
has been c^ducted periodically since 1888. It is the only comprehensive 
source of such detailed information about American families. 

In. the health field, the spddy group focused on two major health surveys 
conducted by the National Center for Health Statistics (NCHS) in its effort to 
meet -the retjuirements of th^ National Health Survey Act of 1956. 

The two surveys are: (1) the Health and Nutrition Examination Survey 
(HANES), which 'involves medical and dental exams and tests and measurements, of 
certain age segments of the population with an 'emphasis on leam^ing^answers to 
specific. health questions; and (2) the Health Interview Survey (HIS), a weekly, 
continuing, nationwide survey of illness pa»tems in U.S. households. , 

; ' In the agriculture field, the study group interviewed people who were 
familiar w^h the public-supporte4 system oi research. It includes research " 
done^by six agencies in the U.S. Department of Agriculture, fifty-five- st^te 
agriculture experiment, stations, nineteen schools of forestry, sixteen land- 
grant colleges of 1890 and the Tuskegee Institute.' These organizations con- 
diitt about ninety-five, percent of tl^e ^nation's public-supported agricultural 



research. .The study group paid particular attention to the agriculture exten- 
sion system. ^ 

\It was impossible to ^investigate every ^phase of the system; therefore, 
the study group decided to pay particular ^i:ention to the agriculture exten- < 
sion system;' Interviews were conducted pijWrily with perspnjiel with the USDA 
Extension Service in Washington, D.C., the Cooperativ^ Extension ^Service at 
Colarado State University and Jefferson County (Colo.) Extension Service / . 
Office. . . , , ' . ' • , ' 

The study ^roup from NAEP did not attempt to compare the educational re- 
search^ system with the research systems of the other fields, noi; did the study 
group attempt to devise a mo'del for th'e^ future developmejj*- of the educational 




reseg^ch enterprise. 

Rather members of the group^did attempt to give a view of what vari- 
ous Agriculture, health and ecoiiomic research methods are like in the hope 
,that those involved with education m^t investigate further the possible 
use of similar methodologies to improve educational research. The descrip- 
tion of the three' fields was limited by the time allotted to the study and the 
funds available to consult with peopjci fiamilar .with the various research 
syst§ms. ' ; 
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. '\ ; CHAPTER 3' 

^ ■ '•' » ' 

IT.' S A MAn;ER OF LIFE OR DEATH 



Almost every, farmer the nation and most processers and distributors of 
food products have used or Tiave.been affected by the results of research con- 
ducted by federal, state and local agricultural agencies. ^ ^ • 

" Price infbrmatlon collected and published by the Bureau of Labor Statistics 
. has been used as a yardstick in updating anS revis,ing wage and salary c6n-> 
tracts, ^social security^benefits, retirement, welfare and other income play- 
ments for millions of Americans. 

The findings, of the health purveys, conducted by the National Center for 
Health Statistics, have been used extensively as the basis fot further medical 
research and in the development of major health legislation. The information' 
: is reported in both 'technical 'and popular publications concerned wit^i health 
problems. „ ' ^ , 

For those who worked on the NAEP study, it was apparent - that matters of 
food, money and medicine hit at the heart of what many people perceive as their 
immediate surfvival needs. ^ ' *» ' - * 

- Tfx^ NAEP sti^y group feel that, as unfortunate as It may be, education 
ranks lower, on the list of priorities of the general public. They feel that • 
this may have been an important factor Ihfluencihg the supjiort that* the^&elds 
-of agri-culture, health and economics have received for their rese^cci^Kwell 
a? the*attention that they continue to commands. * 



The fact that . educational research gets oniy one percei^t oi.th^.$21 billion 
^speut by the federal gov5fftunent on research and development activities indi^ 
^cates that those ^o make decision? about federal appropriations may. not fft^lj 
that educational research is a critical area. . * - * i \,j ^ 

It is ;apparent that the educational community needs to\do ^ far betti^r' . 

\ ' . 

job of contTjtocing key legislators and policy makers that providing up-to-date 
information about educational needs and the outcomes' of schooling cannot be^ 
delayed. They must be impressed' Vith the idea that- supporting research that 
.will lead to improvements in education is as important as funding research 

m 

seeking a cure for cancer, finding a better method j^f lighting crop diseases, 
or determining a formula for calculating wages. - 



CHAPTER 4 * . > 
DOES THE RICTT'.HANP KNOW WHAT THE LEFT HAND. 3B DOING? 



A concern among agriculture, economics ,and 'health research organizations 

is that research findings should be synthesized, placed in the context of other 

/( , ' ' ' ' 

r^s'earch and, whenever possible, not duplicate^ research done in the past or 



currently under way. 

In each of the three fields, the process for identifying research needs 
and determining new directions for resetarch involves many different avenues 
of activity. 



The process includes, but is not limited to: 



~ A thorough examination of -.past and current research-^rovicfed?^ through 
■various data and .information systems. - ^' * 

7- ^ A review of existing and planned fedetal and state projects and pro- 
^ ^ grams . - . , . * 

Consultation with various.^ technical and administrativeTexperts from 
outside agencies and organiza^tions, as well as from within the sys- ^ 

~ , Participant ion in and organization of national,, regional arid local 

conferences where researchers can share ideas with users of the data 
^s well as with other researchers." ' * \ 

— Close contact with private organization^ and other goverWent agencies 
conducting similar or related rjBsearch. - i . ^ 

^ , . ^ \ 

— Communication with ;fed^al- and state legislators through -jpersonal 
contact, attendatice at hearings and examinatian^ of legislative records 

^ a^ reports. 

■ . • ' . ] 

Development of specia^ studies to identify needs^'anS'-lavestigate new 

\ approache^. ^ - ' v 



; • ~ ; .%iveys if - Impact of currejit research on users and attitudfe of 
/ * users about future research. . , ' . iT 

Review of requj^sts for informatdoh abouj: research. 

Activities* for l^ent^^ylng research needs and determining new. dlltectious 

for research alsd caa/b^ bolstered by the existence- of a structure that assures 



comnmnlbatlon b'gcweeti- and among researchers' and research users at various 
levels.' The U\.S4 Department. of Agriculture (USDA) has built such a^ intricate 



-structuc^^ . \ I ' . r 

, Each of the major USpA research agencies and the National Association of 
State Oniver^ities and Land Grant Colieges (NASISLGC) , whose members represent ^ 
^uniyer^ity and college departments of agriculture and extension services^ 
report ^ directly to^ the U.S. Secretary of Agriculture. 

Members of these drganizations and representatives of regional and state 
research agencies form ^ the Agricultural Research and Policy Advisory Conmiittee 
(ASP AC), which makes policy recommendations to bath federal and state. 'agencies.^ 
ASPAC makes its policy recommendation on such matters as the pl'anning,, evalua- 
tion and coordination of long-range, unified agricultural research programs. 

the ABPAC^determines which wo.rk should be 'done by federal agencies 'and by 
state agencies in order *^or the national programs to be carried out* Both^ 
regional -and national agricultural meetings, involving USDA and state agency 
administrators, are sponsored by the USDA and NASULGC through ARPAC. 

Natiopal< a^gricultural research planning committees develop national needs 
and priorities ^f<?t regional research programs* ✓Regipnal needs and priorities'^ 
are set by the regional planning committees that us^ information -from regional ' 
research programs-, vyfeder^ and state research administrators, industry ^oups. 



state and f^^^al legislators'. Within regions, regional' program^ task, forces 



plan' for research on- 3ip.ecially selected topics. - - , * . * 

. i Another feature that .appears to increase the effectiveness of ^agric^lturaj/ 
research efforts is the willingness of an organization *to .amalgamate -research 
eflUprts to avQid duplication^-of effort. * v 

' ' , ' *' . ^ . ' ' ' ' ♦ ^ ^ . 

When* the-mhird cycle Health Examination Survey (HES) ; wfaichj^ concentrated 

pa physical examinations, was* being .conducted, additional^ studies ~ inclu<y^g 

aINational Nutrition Survey authorized by Confess — revealed thak, iH certain 

aireais of the nation and with certain aglp; and income* groups', malAutritidn ^d ' 

^' uhdemufrition were' glaring problems. - - - 'V * 

Iflien it came time' to develop a -fourth phase,, or "cycle^" for IffiS surveys^ ' 

the National Center for Health .Statistics (NCHS) decided it made sense to loin 

. HES and the National i^utrition Survey system into one "dual-purpose" ^survey 

renamed the Health and Nutrition Examination Survey (HANES). They thought that . 

combining the two,- rather than b\iil4ing a new system for a continuing nutrition 

survey, made sense because both tasks could utilize highly trained field and 

headflu^rters personnel whose skills are unique and c&f f iciilt po duplicate. ' 

When it . comes to the responsiveness of researchers to the suggestions 

of. outsiders, another good example can be found in^ the HANES ,e:d^erience. 

ting the planning for ^the jciJst HANES, conducted in 1971-72, Carl 
. ^ . ... . [ ^ 

wipter, M.D.,, director of the National Eye Institute, indicaited an inter^t in 

ootalniM more definitive information on the total preva;Lexlce and distribution-^^. 

o^^specific eye diseases' and related conditiond in the United States. -'He said 

itwot^dl help' the instiWe set goals for future emphasis in' its research ^ 

program^-. * ^ ^ ' ' • . 

HANES res^rchers encouraged two National Eye Institute optff^ 
to develop an examination form and standardized protocol for opthamalic ex- 
aminations that were used in the 1971-72 HANES study. ^ 

14 . ^ • 



. - J 



In generat, the NAEP^study group discovered a spirit of cooperatibrl' gather 



• / ' than competitiveness ^ng researchers in th^ fields of cagricult\^e„- ecbnomic* 
'and health -rv cooperation that encompassed both private and .public institutions 
. and that; ^en connected\with* public institutions, extended across. local, 
s$ate and federal jurisdictions. ^ . ^ ^ . ' ^ 




•4* 



Ay 



ERIC. 



22 




^ * CHAPTER 5 v.. • 
IS IffiTHOD IN THExMADNES's OF RESEARCH 

It waa the observation "of the NAEP study group that ~r in those areas, in 
..agriculture^economics and health where research studies were conducted and ' 
survey data w«re collected ~ that considerable time was provided for the 
•development of both "instruments and method^ogy.' 

In the case of the Health Examination Surrey, development of instruments 
for the survey involves a series of activities over^a period of ''several years, 

« 

During this period there is 'Ixtensive' involvement ^ot experts from concerned 

j~ ' ' ' " ' 

gov^mment^health agencies, university health and science departments and other 

. public and private health organizations and associations in th^ development' 
and'i^eviiw^of the instruments. These experts are not'bnly specialists in the 
fteld^Feing, tested but arealso representative of the people most likely to 
•use the data. '* , . • ' * * ' 



•St . 



Members of the National Center for Health Statisttcs (NCHS) staff care- 
fully examine the experience of other health surveyti "both national and.inter- 
national.- They also visit specific iasti tut ions, and individuals across the 
country in order to discuss prdblems regarding the proposed examination. 

Following development and review of insfruittents for the exanOnation, NCHS 
conducts extensive ^llot testing to check on the feasibility and acceptabUity 
of new examination procedures^ This includes e^luation of questionnaires, 
interviewing techniques and administrative proqedures. More than- one pretest 
is conducted so that there is op'portunittr for redeveloping instruments or 



f 



revising adm i n i strative procedures and then testing tho^e changes again. . 

The tesult oS this careful planning, development and field testing has 
been the cteakion of ext;remely^ creative and complex instruments that ^provide 
the data^inost needed ^ the users* ' * " 

The Successes of the Consumer. Expenditure Survey (CES) in collecting 
minute and extensive data regarding family expenditures can be attributed to 
its intensive planning in the area of data collection. ^ '/ . 

^ / . . • " 

Data collection for CES is under the direction o'f Ithe U.S. Census Bureau's 
permanent professional field staff, which operates through twelve dp^a coUec-^. 
tion^ centers throughout thi United States.^ . ^ 

Additional local program offices ^are esXabliShe^to assist in the collecT 
tion of data. Each of these local- offices is d^uected bf. a Census Bureau super- 
visor, who hires a staff of approximately six intettviewers. The supervisors 
make a point to hire interviewers who have' alr-^dy demonstrated ability in 
previous surveys* conducted by the bureau • 

Staff meniers are thoroughly trained prior to beginning their work on t^^ 
survey. Interviewers receive seven days of classroom training plus self-study 
training materials. Additional home study an^ classroom »t raining are provided 
prior to the- beginning of each phase of- the survey.. 

formal training^ is .supplemented by on-the-job training .to insure job 
performance at a Ifevel of \ established standards. Quality contxpl measures, / 
such as editing replies*, observing iiiterviews s/nd jreinterviewing selected 
sample family units'Ss^re used throdg|iout the survey. 

In an earlier consumer expenditjire Purvey, some testing of collection 
methods -was performed by the U.S. Buteau of. Labor Statistics and by the Survey 
Researph Uboratory of the University 6f Illinois. These testsr'and. the 



experience <?f other^' countries , indicated, that hi$h-qua;idty data could be 
obtained' If the survey design was : tailored so that information on la^er and , ' ' 
more easiiy recalled expenditures was collected by periodic re<»ai. Interviews ' 
and day-to-day record keeping on small, less expensive items'was" Obtained by " 



iialng methods' such as diaries < 



As a result or.thik testing of^co^^tion methods, quarterly and diary ^ 
coUec^i^n techniques were introduced into"-the latest consumer e:tpeiiditure 
survey. -The' quarterly collection of data involve'd the larger and more^'easiXy ' ' ' 
■remembered ^expenses, while the diary involved a daily record -of all expenditures 
for ' two one-week periods. ^ . * ' >^ 

Due to the tremendous iiumber"^f research studies conducted *by the agr.i- 
cultural system it is difficult to detaU the extent o'f planning for instru- 
mentation and methodology. However, it became clear during interviews with 

agricultural personnel that most agricultural research is carefully planned. 

^ . 
As already pointed out, agriculture has a very elaborate system for re-. 

search planning in which most major research -is reviewed at several different 

levels. Professionals from both public an^ private areas are involved* jMany 

of the research studies involve cooperative efforts amony personnel at the 

land-grant universities and colleges and other agricultural agencies . \ ' ' - 

Before research studies can be included in agriculture's Current Research 



"Information System (CRIS) 



, th^^^se 



^search staff of the Cooperative State ^lesearch 



Service (CSRS) must review ank approve the research-. . This. staff contains 
research specialists experienced in ma:|or areas of <:oncem to the people 
served bpr the U^A system who . demand that any wported d^^ represents ^care- 
fully designed and analyzed r^tsearch.. • . 
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>^Th^ aeaibcrs of -the NAEP study group have had> considerable experience with 
. J. ■ ' . , . ■ - • . " ' 

survey and other' ^search studies conducted in^the field of education, i 

group feeis that educational researchers ara. seldom given the funds or time to 
, , ■,■*'.-,•' •*.■•' • •; ..... ■ .' 

. devielop tMe instrtmenfs or data fcolledtio^ mesth'ods o^the complexity or'qualitr 

> • . '/mm- ■ ^ ^ , * ■ ■ ■ ''''y \ 

; found in • tl|e 'Healtji-''Exaw»ination or;^OoMuitt<if E^rfjindi'ture surveys. " When such. 

funds are provid^,^a^ lin the case of 'che^ National Assessment of *Educational 

' " Pr<^gress, the requ^r^ofents |o*. informatiojx are(" so- yast that it is difficult 

• for the orgafiization to do 'in-de^h^re'search/of .sp(|Cj^t{^9pi?s 



:s.« 



On the basis, of the^experience of the study group vith research coriducted- 
in otlier fields, a^^aajor recotanenda^on, or hope educa^nal research^ ' 
would be that sufficient mone^/and tfime bfe orovided to do iu-depth research • 
on critical issuufe^ This would jnSan' adequate funds >and time to cdnduct , 



extensiye planning axid 
methods • - — .' 




lid testing'of botH instruments and data collection* 
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._ ' . ' CHAPTER 6 ^ 

. \ ' :,. putt-in'g your best foot forward 

■What creates high moralfe among research staff members and induces then 
to give tiieir "all" to their work? . , • , v 

Members of ,the health, agriculture and economics research systems, described 
a number o^ conditions that thejr believe positively influence perf ormtoce of ' 
research staff members. ^ " ' * 

They think it is important for staff ^nembers to research organizations to 
have divefrse professional experiences that allow them to understand all, or 
most, aspects of their research system, not just the sector in which they are 
working. . , 

For instance, if a research scientist formerly has held an administrative 
post in the research organization, or If an administrator formerly has been a 
researcher, then greater understanding cian exist between the two, making cpop- 
' era tion" easier and their working relationship more satisfying. 

The NAEP study group also was told that it is important for research pro- 
fessionals to have a personal stake in the outcome of their. work. People inter- 
viewed in the agricultural^ economic and health research fields sAiL this can 
be accomplished in a^unijer of ways. ^ * 

These include allowing researcH'ers to cO-author popular articles about- - - 
tli^r, firings along with a prof essignal writer, giving researchers' the time 
and,. money to' develop new research methods , and improve old ones, and locatiljig*'" 
r'ese|ttchers whefe they can see their findings turned into practice — and even ~ 

assist in that process. A ' . ^, « ' 

• — . • 20 / . . 

,,• ,■ ^ . -■ ■■■ . • 



The study group also was told that staff ootale is imptbved when employees 
know exactly how ideas .and information flow w^ti^in^' their research organization. 

Another conditipn described as aiding staf/ morale and performance was 
giflng employees an op^rtunity to travel apd take leave tim^/^discuss - their 
J *^rk with people outside their own research organization^ . 

Several people interviewed said that if researdj^s know" how to and can 
Influence decision makers and policy makers with ^heir research fadings, than 
-they xrlll get more satisfaction out of their work. 

Finally, it appeared important that research organizations make a special 
effort to use their own staff members whenever possible rather than to rely on^ 
outside consultants vho often have little at stake in outcomes of the research. 



CHAPTER 7 

^;t'| everywhere: it's EVERjnraERE: 



\<rm|c — - ^ 



OR- IS IT? 



^ Onc^ research data are gathered, they, are of little value if they sit on 
a sjielf gftthetiii|^ dust* An important step in the research and devel9pinent pro- 
cess" is widely distributing^ or disseminating, research findings. . ^ 
'o Later in this report, there will be an ey^ainttaatlon of the crucial step 
beyond disseminatiqn, ' that of getting research adopted into practice. But for^ 
now, the focus is on the technology of distributing the results of research in 
the agriculture, economics and health fields ♦ * . ' * - 
Generally, it was di3^overed ti|at -these disseminationv tools are used: 

^ " — Easily^ accessible computer information and retrieval systems that 
^ ^ people are trained to use. - - \ ^ 

— Easily understood flyers, single-concept publicatiops, short .semi-^^ 
technical articles, newsletters and tdt^t sheets* 

^ Key reports indicating trends over a limited time span on vital issues^ 

— Comprehensive reports confined to a particular study or survey. 

— Special publications of research produced in cooperation with private/ 
industry or other government agencies. 

Articles in popular magaziiies and professional Journals, sometimes 
* co^authored by researchers and professional writers. 

~ Regular contact witfh media people. - 

^ T- Personal contacts between researchers and potential users. of research. 

~ ^ Testimony t^efore lejgislators and task forces. 
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^ ~ Mlcrdfiche, microfilm and microdata tape transcripts providing 
" technical ^ata. o - • - ♦ 

— Low-cost spots, on television and. radio (often public service programs) 

- • * ^ 7 

— EiAibits in pub34c areas and displays and presentations at pxofes- 
sional jaeetings. ^ 

' ' . : . ^ • . ' • " '/ 

— Films, slides and videotapes providing visual .contact with reseaf^- 
, ers and their work. ' . ' 

To offer further details on dissem^ation processes, her^ is a look* at 
two approaches to distributing^ research data, one used by the National Center 
' for Health Statistics (NCHS) and the other by the Consumer Price Index (CPI). • * 
oiffice.' • ' ' ' ' 

There were two techniques utilized by NCHS in reporting fin4ings of both 
the Health Examination and Health .Interview Surveys that particularly 'impressed 
the NAEP study group. ; ^ 

Jean Roberts in the^ Division bf^Healtli Examination Statistics indicated 
that in preparing reports of .survey results; the division often contracted with 
individuals outside the center to assist division staff in planning for the 
analysis of data axid the writing of the reports J These indivl4Ti|as were 
experts in the area being analyzed and usually represented -the type of person ' 
who would most lik^l/ utilize. the results whifen -they were published ! By having 
these experts involved in the analysis* ^ab well as the writing, the division 
was assured that the data would, be summarized and reported* in a 'form that would 
be appealing to some of the major users of th6 data. 

. ^ - : t 

NCHS also publishes the major research findings in a seri^ o^f reports ' 
entitled Vital and Health Statistics Series . While certain reports deal with ' 
a^x overall description of a lafg^ study, the majority of the^reports deal with 
th^ descriptionjDf a particular phase of a study or a well defined topic. Each 



of .the reports is^nuabered and color-coded according to major areas or studies. 
Most of the reports, present a.sunmary of the infonjiatiroor findings provided 

* \ ^ . * ^ ^' \ . ^ ' 

la the rept)rt In the beginning and then proceed to more detailed descriptions 
H of tiiie data or Information. By presenting reports on clearly defined subjects, 
,^tii a summary of the findings in the beginning of the report, a reader without 

a;spphlstlcaj:eii.lcnow3:^g€^ of health information c^n quickly locate information 

' . : ^ ^ ' ^ X' ^ \ . " ^ ^ X . 

on a particular area of interest* 

The CPI Ms, released once a" month. The day before the release, special 

• reports tailored to specific Interests are sent to the chairman of the Coimcil 

of Economic Advisors who passes it on to the President jand other council 

members . The CPI ^reports also go to the Secretary of Labor who passes^ informal 

tlon on to the Commissioner of Labor Statistics, who is responsible for testify 

, ing on the CPI before congressional committees and the Council of Ejyjnomic 

Advisors^. • ' 

,.- ■ • . , - ■ 

^ On the day the monthly CFI report comes out, press releases^ go out on/ a 

rigid schedule — .8^30 a.m.. Treasury Departments Office of Management and 
Budget, Federal (Reserve BoarS; 9:15 a.m., concerned congressmen and select^ 
, nfembers of the medl^ and 10:00 a.m., the public ancf people on^a mailing list. 

The monthly CPI report provides th^ telephone numbers of Bureau of Labor 
Statistics staff who can answer questions about the report. Questions from 
legislators,, government agencies, business officials and the press are usually 
handled by the director in charge of the department "responslbiii for the .CPI 
report or selected members of the CP^ sta^. / V 

What is important about, the CPI data release process is that it is 
designed so that people and dgei^cles who will make major legislative and politi- 
cal decisions based on the CPI receive the* data before the press and general 



public. In this way, if there are particularly unusual findings', these key" 
people arid agencies can explore the data further before they have to provide 
answers to qu/Bstions from the press or the public. ' ' ' 
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* ' ^ ' - CHAPTER 8/ ^ • * 

• *:A FRIOD CALLED CRIS ' \ ' 

^ . The NAEP study group^jeel s^^jih^t among the many research dissemination tools 
it examined, ^ computerized infotmation and retrievaivsystwi of the public 
agricultural rese^irch communitpy is wortljy of. special mention. It is called- the 
Current Research Information System, or CRIS "for short, 'and became operational 
In-^lSeg. It is a research project system. providing for the interchange of ■ 
information 'among scientists. ^ ' / 

•The, USDA views CRIS as an "essential element" for. effect^ive research ^ 
planning and coordination. . ^ . . * 

CRIS includes infongaticjn on projects for all research conducted by the 
mammoth, publicly-suj^por^ted agricultural' research system, ' This information is 

- updated annually, ' All researth pj::pjects are reviewed at least every five years. 
The NAfiP study group fol|id\^^^ evidence" in Interviews ^that CRIS is well 
understood' by professionals atj ev^ level in the^enomous agricultural research 
cpmmunity and is heavily used. The ^roup further determined that by its heavy - 
use, the professionals d€^em it a Highly Effective tool*. v > ' 

^"Agricultiare , people really know their data system,"- one of the NAEP rf&- 

< • ' ' ^ _ - ' I, f - " ^ 

' searchers reported. "They know it) through use ~ constant "use- ^Che norm is 

^ ' , " I 1 < * 

to use it and it works," he ad^ed. < * " ^ \ 

1 M . : V' 

The NAEP study group also fou^ifi it significant that- CRIS provides all its 
■ data to another -ittfoifaiatipn system, ^ the Smlthscfaian Science Information Exchange, 
thus^further landing the availability of the data. ' , a, . 
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Inf oCTiation 7««^rches through jCRIS are provided free of charge to employees 
of'USDiit agencies, the^state agricultural experiment stations, state extension 
services and to "cooperating institutions. CRI5 encourages these .prof e^slonals 

■ • V ■ 

to use the easiest method of requesting information possible. 

Nopnally, CRIS completes a search and returns requested information within 
two weeks . * , 

-CRIS tells the user the location of the research project#and the organize-, 
tion responsible for the reseaorch, identifies the investigators, g:^es dates ^ 
for the beginning and ending of the research ^x^ject, describe! research • 
approach and provides details of annual. pro grass reports on the^esearch pro-' 

* As alluded to earlier, the USDA actively promotes CRIS among\rotential 

scientist-users, l^^^^rs are sent out periodically to state an<i federal agrir ^ 

cultural^ scientists encouraging them to use CRIS,. Among other things, USBA 

tries to persuade them that by using CRIS, they can more readily identify other 

scientist's that fchey would want to talk with at professional me^S^l^s, 

Thus, CRIS not only provides information, but also encourages further comr " 

♦ 

m u nl catlon between researchers. y 

CRIS is also available now, for a fee,, to researchers i^Q the United States, 
Canada, , Australia, Mexico and Western Europe, who are not among the [participat- 
ing federal and state research organizations that get the service free. ^ 

An interested apientist outside the itate/federal agricultural research ^ 
Item can gain access to CRIS through any make or model computer terminal th^t 

4 

^ can ,be used for what is known as "dial-up" service. 

*The easy access to CRIS data, Che encouragement of its use, the merger of * 
CRIS with another information and retrieval systepT^Cthe Smithsonian \s ) , the 

' , . ' 27 " ^ • ' ^ ; ' ' 

ERIC ' ^ . ^ 



axmual updating of CRIS data and ^ the review of research projects described in 

CRIS at least every fi\« years, alL appeared to contribute -'to . its effedtiv^ 

\ ^ ' ! ' 

nests and its positive image in the agricijltural resil^rch' community • ^ \ - 
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CHAPTER 9 s* , • 

DID YOU REALLY SAY • ' , ' 

"INSTALLING ^ IfiDNITORING CLUSTERED ^PiWDDUCTS IN DEMONSTR^ION SITESt" "''^ 

Every research organization should have least one Edw^n "Newnan on Jjoard, 
someone to turn gobbledygook into imderstandablc prose. -From their respective 
institutions, researchers and policy makej^s, often fail to" conm^nicate* well ^ . 
\^ with one another because they speak different languages. , The problem also " k 

^sxists between researchers and people , trying to put researchT-into practice >in. 
* the fieli. . - • ^ : i . . \ . / 

_.0f cpncem to t^ NAEP study gtoup is how . researchers in education can. ^ * 

J ' ' ' ^ • ^\ ^' ^\ 

bridge the coomunication gap ^ so that they can influence iJnportant policy makers 

as well as have an impact on teachers and the public they serve. » " , 

' > ' . * ' - ' ' 

The effective use o*f language is a ke^ to Success in any prganizationaJL 

Ok . . 

research and development enterprise. 

In the field of agriculture, particularly, the KAEP stuiy group sfw tl^e 
importance of a variety of comntunication devices —.brisk fact sheets; terrf^ 
newsletters; typographically pleasing, easy-to-read pamphl^s often concentrat- 
ing on a single concept; ^an^ bright* condensations of research results ti^ni ^ 
reserved compl^Lcated t»bles f d» attached 'indices . * ' / 

All appeared to- pay off in getting Results frcm research/* \ . * ' 

The USDA publications studied range from as few as two to as many as ^twenty 



/ 

pages in l^engClT and all used^helpful illustrations, soxoetrimes 6^ every pa^e-. 
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Numerous footnotes and other notations 'guide the reader^to tore technical, di^ ]' 
tailed infoi^ination as needed, '* 

^Jhile piJblications of CPI.and. NCHS ^ere often heavily statistical by com- 
parlson and need a'^ocabulary at the high school level or above, researchers 
in those two fields well as those in agriculture^ proved' capable of translating, 
research resjxlts for slegislators and other policy n&kers,* 

They produced statements, perhaps as brief as one jor two pages, that de- 
scribed the significance of research findings, provided key st^tisftics, sug- 
gested several alternatives for action based on the researcli^ results, indicated 
both the positive 'and ^negative implications of 'the ^ata and mentioned other 
, research that would potentially suppprt or conflict with their latest findings. 

The N^EP research group concluded that from ^the standpoint of communicat*- 
ing with both policy makers'" and practitioners in the field, the agricultural ^ 
research community has the most well-develbped model of the three disciplines * 
studied an^ might have the most; to teach the educational research community in 
flight of the variety of potential users each system should be reaching* ^ 



/ 

/ 



/ 
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; . . CHAPTER 10 . • ' . : 

i ' ' J- ■^ . • . 

; J ' ,^ . WHO^S-YOUli NEIGHBOR? * , 

Question: If you were a leader, in 'educational research, would 'it be a 

* • * t * 

good idfta to buy a house next door to a congressman? Answer: Yes, provided 

'you knew how to interpret your research results and size up^their policy 
* • . * * • 

'ini^lications. , - - , ' ' 

Livb ^ext' <io 




Of course, one does not have to* literally livb ^ext' <ioor to the powers 
that be, but in. talking 'to researchers in the agriculture; ecouQinics and 
health fields, the NAEP study group often fo^nd a significant relatlonsh 
cween th^ geographic location of their offices and thos6 of key policy ma! 
In their fields. ' • * 

Th^ group found it helped for researchers or their representatives t6 
work near key decision makers^ Their close proxlinlty led researchers into 
learning how to communicate with decision makets; and it^ allowed for constant, 
regular contact between the tw6 groups. ' ^ 

Foi: influence at the federal level, maintaining offices in the Washington, 
D»C* area seemed practically essential, "the closer to the Capitol, the better^ 

The NAEP study grdup foixnd', for example', that key people with^^the Consumer 
Price. Index are in constant contact with what ^hey call '^critical staff mem- 
bers" in the U.S. Congresl. The CPI director is called by the head of the 
Federal Reserve Board When he is faced with an Important .decision that could-te 

influenced ^y CPI research .data. 

* . _^ 

The USpA has a congressional liaison officer working on the Hlli; .regularly 
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providing neeied^^cesearch infonaatiou to Congressmen and their -.^aides. Addition- 
.ally, USDA keeps a steady stream of research experts flowing' into Congress to ./ 
testify at Imgortknt hearings. - '\ ) ^ , 

la- the 'Educational research community there appearis' to be seme growing* ~ 
awareness of the need to bolster^ efforts to influence policy makers — to hi ^ ^ 

in the .right pyacft at the right time and more particularly to make regular 

■ - /" ' "^^ ' - , • - ^ ^■ 

appearauces at Congressional hearings. ' . " ' ■ «• , 

Roy Forbes," director of the National Assessment of Educational ProgTel*rr-_ 

• ' '' t ' - ' ' ' ' 

• one of the major -ed^ucational research efforts in the Unit.ed States^ recently 

said that he realizes more and more how important it, is Vo have educatfpnal " 

research interests actively represented in 'Washington by key experts ; articulate 

-, . . - • ■ ■ 

constituents or liaison officers who maintain close ties with Icey dfecisioa 
makers. - ' ' ■ ■ ■ ' " ' : . ' • ' 

. /. Peggy Campeau, a'' representative of ^ the American Ix^titutei'ior 
Research in behavi(iral "sciences (AIR-) , after participating, in a seminar on 
Wa^^ington policy making this spring, came to rjalize that' it is important fot 
the highest-ranking corporate officer to speak on behalf of AIR at any Cdn^res- ■ 
sionaA hearing where AIR might want to make an impacts ' , ■ 

"Anything short of that... will not carry t^eigkt" w^^ith decision, makers" on 
the Hill," she concluded"^ " " ! - ' / , 

.The Institute for Educational Leadership, a program of the peorgfe, Washington 
.Ctoiversity in Washington, D.C., has .been actively seeking to strefligthe&present.;.^ 
and potential leadership in American education at" the policy . level -6f state and 
federal government. lEL'^ programs .are designed to assist '"policy makers in' ' 
i^Bp ^ g mmunlc^tions wifch educators by 

creating new i^brums fot- the^inter* 
change of ideis among federal, state -and local education policy makers^/ 




/ Jacob J'. Feldman holds a-po,si/Cion in , the Jiealth. field' whicrh the 'NAEP study ^ 

; V group beULeVes helps influence pplicy based on research*- ' - 

'Feldoan's title is associate ^director" ofV analysis, division of analysis,* ' 
National Centex?*for Health Statistics. Anong his 'responsibilities are. running 
a clearinghouse for health statistics, stimulating development of measurenenfc " 

• ■ • ■ ■ ^ ^ ■ , . f . ■ ■• ■ ■ ; V 

^ concepts,' preparing an overall plan gpr the analysis and presentation of re-^ 
> search data^and preparing afmual reports.'" ' , " 

But ^ihat the NAEP study group found ^st interesting ainoij.g Feldman V myriad 
tasks is;tTie preparation of repori;s to b'^e used in Congressional ^stiiony or 
for stJudy by^appropri^te^subcoinmit^ arid other key, decision-making bodies. - 
*These. reports' are a synthesis of studies dojxe by a nuai)er of /health researchers, 
- His worl^ establishes a strong J[,^k between research\nd,-^te6^ii^ Mking. 

^ . ,v . ' 'Alsd Vort^ special note is the- keen polit4.cal savvy of USDA researchers. 

^ VSHA. officials Will draft legislation upon request for cdngressionil aides 
. , whether or not the ^gislat ion's intent is supported >y -USDA/ However, later 

they are cateful to' avail themselves of opportunities to express thefr feelings ' 
about the proposed legislation during the* regtilar l^lslative review procedures. 
A rather unique model of how to influence polipy decisions with, research 
^ Was found the Health Policy Program, Schoc)l of- MedieiniB, University pf'- '- '' • 



"''^tifomia, San Francisco. 



J'^fS^' "^'' Established in 1972, "the Health Policy Program's objectiv^ are to iitprove 

the qiiality and flow of informatio^i to those who make heal^ p^icy /decisions ^ 

at' the local, state and national levels. - . : - - 

"7 . • • ■ , ■ 

During t^e regulat academic year^eminars on poli'cy issues are held and 
lAclude representatives' of the faculty of- other -nh^rsitied, the^esearch 

staffs of state and federal legiatlators^, staffs ,of professional assbftiStions 

./•■.;.- 

1 . ' ■ ' 

3*, . . . ^ ^ 

t " ' - • 



f^^:^- . 5 lljte. the American Society of Interjjal Medicine, and members of, such orgahiza- 

i*.}"^ r ■> ■ ■ . - ■ ^ • • ' 

tlons a^' the Rand Corporation. . . . 

The Health Policy Program also has a "Washington Study Group" which works 
.to.etdiance comnmxiication with federal health policy makers and' their JissistantSe 

The progr,am engages in analysis of data that relates to health policy . 
issues and' of the current policy-making process. When" information is not avail- 
able to 'Shed light on a current pollcjr problem, then the program initiates 
academic research studies to. fill the gap. 

Members of the program attempt: to give assistance in both formal and 
informal ways to policjr makers in Congress , the Office of Management and 
Budget, NEW, the Veterans Administration, the California State Senate's Ofiice 
of ReseajcJ^and other organizations. 



This section has attempted to show procedures for maintaining close ties 
between researchers and decision makers. Working within close proximity qf^^ 
one another or^^es^blish^ing a^formal- network for regula^y getting togetlifj^ . 
wej^ the* primary methods encountered^. • ' ^ 



CHAPTER 11 



J 



"shall we- consider adoption? 



Irv Lehmaim, a Michigan State University professor, suffered a coronary . 
in May 1975 Before the attack he exercised infrequently and ate "as he 
damned well pleased." ' ^ . ' 

But the heart attack brought Lehmann to grips with reality; He knew, he 
had to change his diet and exercise regUlarly. 

In the course of trying to find out exactly what to do, Lehmann encountered 
data based on research done by the National. Center for Health Statistics!." ' 

..-The research findllngs were contained in a • popular magazine article he read 
on how to prevent heart attacks. The data also^ was the iasis for- pamphlets 




supplied to him by the American Heart Association. 
^ ^The research findings — translated into layman *s language ~ su^iported 

what^^Lehmann. s *pfiysician had told him.* He was persxiaded. He was impressed. ' 

The information has helped to shape his hibits .since that day li May two years 

ago. Now he jags four miles a day at the .fate ti 11.5 minutes per mile and 

^eats a low-cholesterol, low^fat diet'. 

, Shirley Weddle of Jefferson County, Colorado, a home economist by train- 
' ing, because of membership in a prqfessioiurl organization, recently found at^ * 

opportunity 'to learn about food preservation techniques through the agricultural 

extension service in Jeffirson County. ' , - ' ir ' 

She agreed to give fifteen hours of service* in exchange for fifteen hours' 

of training. The training, based on USDA resedtch,^ incl^ded reading- handouts. 
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tchlng video tapes and, attending^^workshops run by agriculture specialists. . 
freddle cane away with a good impress j,on the information was clear, not 
. too technical ~.and she had an opportunity to suggest some things herself. 
'How she^s able to trajji others *in food preservaj^i'bn ,and get continuing educa- 
tion credit for'^^her efforts. And a[s* a sid^^nefit, her children fa^ve beVome 
interested in nutrition and preserving food. ^ * \ 

In short, the IISDA research findings on food preservation have'touched 
Weddle's life and thai of her family. ' ^ ' 

Johi^ Smith (not his >real name} is getting a^ divorce. Smith, ,of New York, . 

* * • * 

is facing a large ^ilimony' settlement. He desperately wants to know^iow^TEnflai- 
tibn and cost-of-living factors are going to affect his paymeitts, which in turn 
•will affect the quality of his life. 

^ecetttl^^>e^alled Pat 'Jackman at the ^Cohsumer Price Index^off ice in ^ 
Washington, D.G. Smith wanted some information based on sound ipes'earcH. As 
it turns out, CPI^'data will be introduced in the divorcju^cfutt proceedings and 
will help, shape, the, financial future for John Smith and the ex-Mrs. Smith.- . ^ 

These examples illustrate how, data from the agricultural," economics and 
'health research organizations are affecting the everyday lives of everyday % 
folks.^ The, research '-results actually touch people. * * 

In the jargon of the research world, "adoption" is the process by which 
- the results of research changes 'the habits or practices o£ people. / c 

In the case of Lehmann, the adoption was ^accomplished through popular ^ 
/ magazine writing, non-techndc^al. pamphlets and talks with his physician. ' Tor 
Weddle, adoption occurred bUcause of publications, video tapes and personal 
contact with OSDA workers . For Smith, the CPI data was of , essential use because 
• it has over a period^ of tim^ been built into manyie^gomic formulas* even into 
alimony payment formulas. - ' . 

\ . • • . ' - ^ _ y 
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^ • CHAPTER 12; / 

: • • T " , / * ^ * 

ADOmON THMJUCT LINKING " ^ 

The adoption of new practices 'based on research does not jixst happen .auto- 

. laatically. The research find:togs should .be distributed in an^easy to use, 

• ' " ' *■ » ^ * 

lAiderstandable form and often there^has to be an agent working to lixik the' re- 

search td the potential user. Many times without this linking agent, research 
d^ta remains just so much verbiage stacked lii a pile on somebody^s desk or' 
stored in a computer. - ^ 

Ia the publiclyrsupported agricultural research system there is. a strong 

link between research and practice that nas succesafxxlly bound together the 

^, ♦ * 

. .two, for many -years. ■ ^ . , ' " . ^ 

^ -<"It is the Cooperative Extension Service of the. United States, which has 
been operating for more than a century as -an educational agency of the TJSDA* 
^ . It Involves interaction among Icmd-graht c6lleges, universities and county-and 
. city*govemmet^ts. . ' / * " ' 

It is a formal system for getting the results of USDA^ research into 
^ , practice. - . ^ ^ ^ 

The scope of the program is -overwhelming. The Cooperative Extension Ser-" 
Vice involves about sixteen thousand professionals, ten thousand support-staff 
^ members, eleven thoiiisand para-pro f'edsional aides and pver a million volunteers. 

. . ^ . .... 

Funding for the program is diverse and^ hinges on no single soujxr^ Total- 

* » * • 

Ing more than $30d million annually. Cooperative. Extension Service futxds come 
from*federajVsta.te, local and ^private sources. 



.Extension services are far-flung. They are offered throlTgh all fifty- two 
land-grant, universities and college?' with offices in nearly every county, and ' 
large metropolitan area' in the United States. ' . ' ' 

•The programs of the extension service are educational in content andf'method 
rather than regulatory or fiscal. 'They are a vehicle for transferring a bro'ad ' 
spectrum of agricultural research findings to clientele in all counties. 

the Federal fixtension Service (FES) serves as a- central office represent- 
'Ing the Cooperative Extension Service. It^^provldes national leadership in ' 
extension programs to insure their coordinated support of significant national " 
goals growing out of USDA research. FES approves the appointment of each state 
dire<?tor of the Cooperative Extension Service, who is selected by the governing 
body of the land-grant college of that state. • . ' < ^ 

While nfejor goal3 for the extension activities are set at the federal « 
leviel, thfere is wide latitude to permit the states tJ[ adapt their programs to 
vaiying needsT * ^ • • . . ^ 

The extension services are admin£stere4' by the land-grant universities • 
through an extension Erector and provide inform^, noft-cxedit education for 
all ages. In the operation of the extension servicej^^there-^^ flow of 

coiaaunication among USDA researchers, extension'prof essionals„,residetrt-teachers 
in the state university systems and researchers of other private and public 
agencies. » ~ ' 

This . conmtunication is achieved at tli^ -university level by having research- 
ers from USDA and other, agencie^ sharing teaching;responsibilities and working 
cooperatively on federal, . state and local research projects with" resident 
researchers . , - 

J 
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Cooperative ejctenaion' agents, in cooperation with extensiofi specialists 
from the university or college and researchers -from the state agricultural ' 
• experiment .stations,- work' directly with fanners and other community people on 
a county level. . These agenta provide .advice ,to .their clientele on a vast array 
of both urban and rural problems. The. agents also aid their clientele by 
giving demonstratisfis, workshops* and short cou^^ses to introduce new and proposed? 
practices developed by research at federal, state and local levels. ^ 

Local community residents and organizations. have considerable, influence 
on^the work of the^e 'agents and the local semi-autononojUs extension offices* . ' 
from .which they- operate. Local governments "as well/^ federal and state gov- 
emments provide funds for various ext^ion pr^p-jects; local Farm Bureaus, 
schools, service clubs and other private and, public organizations work coopera- 
tiveiy with agents on various projects, and local residents are often represented 
on committees and boards that advise local extension offices regarding community 
ne^ds* 

It is this federal/state/l<ical network that ultimately links the prac- 
titioner, the farmer or the urban resident, with an agent knowledgeable' about 
res'earch findii^s'and capable of teaching their application. 

• The NAEP stilc^ group found the linking processes between practitioners 

- • \' • , 

and researcher? ±a the other research systems examined, far less developed or 
extensive than the precedes utilized in agrictature. However, the other 
research c^pganizations did attenipt to .provide some direct s<Axrces for their 



clientele, . > 

Within the Bureau of Ubor Statistics there is an office that provides ' 
technical assistance in developing Consum6r Price Indexes for stfate and 'local 
. ■ governments. The decision concerning which states or local governments would' 
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receive such assistance /are based on some of the following criteria J 

^ — • How politically or economically <iritical is the situation; for 

example, the Biireau of Labor Siuitistlcs provided assistance to the/ 
city of Fairbanks, Alaska, in developing price indexes when the con- 
, stxuction of a major oil pipelJLqe radically affected the ^bononty in 
;^ that area of Alaska. ^ 

~ How useful- will the data be; i,e^, will .the assisi;ince result in a - 
major use of tbj^ data by^the state or local go^mment ant impact on- 
' a substantial number of people. ^ ^' 

* - ' 

~ 'How much staff time will be required, with^a limited .staff and a 

prianary respoxxslbility for producing the CPI every month, the Bureau 
, of Labor Statistics must carefxily consider any expenditure of staff 
. * time. . ' \ ' 



^ What additional costs - are entailed in providing assistance; in* many 
cases the BLS would fxpect • the outside^ agency to coyer or share in 
the cost? of producing any special tabulations. 

The Consumer Expenditure Survey, (CES) d^artment^was also 'willing to pro- 

vide special tabulations for state and local agencies as > well as private industry 

following criteria similar* to those of the CPI people* . ' • 

. In response to the need for more adequate health-related data to ijieet ' 

national, st^te and local needs, NCHS in 1970 began the developnifent of a 

Cooperative Health Statistics System (CHSS). This system was ^ designed to pro- 

* • ♦ " 

vide a wide range, of us^rs of health information witji comparable data, in^ various 

health areas and to provide the analytical. and technd^l assistance to make this 

data useful. The system would. also establish a coalition among ^tional, 

f • > 

state and local data systems for purposes of coordinating the, collection of 
health data. 

•In 'persuit of these objectives, contracts are presently being kwarded by 
the Division of Cooperative Health Statistics System to state and local agencies 
permitting these agencies to determine the most feasible organizational s^truc- 
tures and., content In the collection of health data and to develop training 



40 



.*t»terials that demonstrate current data systems aild provide itifonnation pn 
how to 'Modify ctjrrent systems to meet, local needs. 

As of . Septenier 1976, forty-five states were inv9lved in some component 
of th^ystem and fourteen of the forty-five states had considerable coordina- 
l activity usually in the form of a state health statistics center^where 

direct responsibility could be taken for the collecting of the data for one- or 
mor^ of the cooperative system components and providing consultation to local 
.datl users. . ' . . , 

The educational research community is aware of the successes of the agri- 
'^ACv;c cultural researcli pla nin g system and> the cooperative extension modele. 

^ Richard K. Helrig, director of the National Dissemixxation Leadership Pro- 
ject, a former teacher with a background thfe field of agricxilture, told t^e 
NAEP stjidy group of his belief that an educational professional counterpart of 
the agricultural e^g^nalfi^n agent can make a^ difference in seeing that educa- 
tional research results are used*. - . . . 

.From his e^erience, helrig belieyes that' to be successful sucTi an educa- 
tional, facilitator program must include: money for program development, fuid- 
ing from.various sources, -^invJlvement by state departments of education,/ a 
system for measuring the eff^tiveness ofggfe facUitators, and a focus on 
workihg with that- sinall percentage of school personnALjrtio are most prone to 

/ ' V 

accept change and adopt new techniques, called "early , adopters." ' 

While small ,i^ scoiie, there have been recent attempts to link educational 

research with the practitioner. - 

Currently the Kansas Educational Dissemination/Diffusion System (iOEDDS) 

provides a trained retrieval sjtaff of thirty in regional centers who submit 
ft .„ information searches to the national and local data basis, plus a rftaff at the 
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Kansas State Department of Education who search the consultant, practices; 
programs, and processes files. The KEDDS^ regional centers are located in eighf^ 

Kansas colleges. and universities and in one educational cooperative.. ,KEDDS 

* ' ' ' ^ * // ' * ' • 

also provides a staff of 3ix professional facilitators who serve a% helpers to 

local school districts in developing a process thafr'wiir help them ixr implement- 
ing ^new educa'^ional programs. , ^ " . " ^ • » , 

Amdng the services offered by- these facilitators to any school districts 
in the state. of Kansas are: local needs 'clarification,, organizational . con- 
si4ting, diffusion planning with Title III/IV projects, workshops -on dedfeion • 
making,, and assistance in training, finding, resgurces, ^planning *f or ^change, 
and developing community support, 

/ \' ' ^ ' ^ 
The .National Institute f or ^Sj^ucation (NIE) ^ within the past few-vears 

greatly extended its strategies for improving dissenCLhation of educatiotm in- 
formation between federal; state and local agencies. Included in th^se/strate- 
gies are the following activities: 

^ ' — Awarding o^ gr^ants to state education agencies to d,evelop. comprehen- 
sive dissemination capacities, ' * . J\ 

~ Improvement of Educational Resources Information Service *p3!g[C)l^ *a 
clearinghouse responsible for acquiring and processing fj^rdata base 
input significant educational information. 

7- Fostering of communication' among state , education Agencies and NIE / 
through regional and cCational conferences. ' 

~ L inkin g of resources of- state education agencies^ regional labora- 
tories and centers, and institutions of higher* education t6 local ' 
education agencies and, school sites to work cooperatively on educa- 
tionaj. problems. ^ . ^ 

— Develo^ept of systems by which educational research, and development 
center^^fnd laboratories can communicate jresults o^ 'R&D ,to educational 
^ , practitioners and can obtain input from local education agencies 

.regarding vneeded research, ^ 
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^ Encourage development of papers«and studies that wlXl further aid 
understanding of processes and systems for usix^g and producing < 
knowledge • ' » * 

At a M^rch 1977 conference in Columbia^ South Carolina, sponsor^-by the 

-National Diss^ilnation Leadership Project, , Herbert Lionl^erger, a rural 

sociologist from the University of Missouri and well-known expert^ on a^ri- 

cultural dissemination, spotlighted a^key difference between the 'agricultural . 

and educational research systems and their link to praptitionerS in the fieltl. 

He said that in agricxjilture as well as medicine, innovations ^ame from the 

scientific community. Thus, the agricultural extension agents link the scl-*^' 

♦ 

entific community with the farmer. ^ * • * iS 

But he said that in education, innovation's are developed more Jnd more' 
locally and the U.S. Off ic^ o£ Education's diffusipn network was set' up to 
help spread locally developed innovations. Liopbergei;^ suggested that e'duca- 
tional dissemination should make more use of researchers and universl.ty experts 

Some debate did emerge at the conferet^e f rom?participants who ' contended < 

. . . . *' . / V > 

that innovations develbped locally are not' inferior to those developed at, the % 

^university level. In fact, they argued that st^^ locally developed innbvations 

gain greater credibility '-^ng tho^e who mifi^t^adbpt them because they are 

viewed as workable. ' -it ^ • • - • ' 

The dlssemln)^tion experts at the conference su^ges^ted that the important 

thijig is that state departments of education gi^ ^Qi^iethlkg started to Ij^'k 

research and practice. ^ ' ^ ^ 



The conference 'partici^)ants concluded that cQ|^p{unicat ion between ^and among 



the states ts vital to establishing widespread educational linking mechanisms < 
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AGRICULTURE INTERVIEWS 



*R.J. HILDRETH 



Farm Foundation 
Oak Brooke Illinois 



Managitig Dljcecyior 



BENJAMK blankens 



, EUGENE M. FARKAS 
ELIZABETH S,, FLEMMI2IG 

.JOSEPH GOOD 

OPAL MANN 



JOHN T. MYERS 
DORRIS W. RIVERS 

E. DEAN VAUGHAN- 
FRED W. WOOD 

* c » 

MASON E. MILLER 
DAVID J, WARD " 



c 



U.S. Department of Agrlcuiture- 
Washington; D.C. " 

DlrectoV, Information Economic Research Service \ 
Division 



Director 



Agricultural Research Service 



tnfonffiatlon Leadet, Extension Service 
Home Economics ^ 

" J ' ■ 

Director, Pest bfanage- Extinslon Service 
ment Programs 

Assistant Adminiitra- Extension Service 
tor. Home Economics 

Office of Operations Extension Service 

Agricultural Extension Extension Service 
S.peclallst 

4-H Youth Development Extension Service 

Public Policy Special- Extension Service 

1st . ^ . ^ , 

Director of Information Cooperative State Research 

, > Service 



Research Planning & 
Coordination. 



) . E . C . 



Colorado State University 
Fort Colling^ Colorado 



^ KENNETH OAKLEAF- 
LOWELL H. miTS^ 

*Interviewed b.y telephone 



Assistant Director 
Director 



Community Resource Development 
.Cooperative Extension Service 
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PHYLLIS WORDEN 
WARREN TROCK 
SUE AND^ON' 



AGRICULTURE INTERVIH/S (cont.)^ 
c ' * * 

Colorado State University (cout.) ^ 

> Assistant Director Cooperative Extension Service 

• Director ^ Department of Economics 

Administrative Clerk<^ Department of Economics 



V, 



JEFFERSON COUNTY^EXJ^ENSION 
Golden, -Colorado 



C. IVAN ARCHER 
ANN SAINT-DENIS 
ROBERT CZERNIAK 
CRETCHEN FOSSE 

( 

SHIRLEY WEDDLE 




Youth 

Hone Economics 

Community Development 
• * 
Commtmlcatiohs^Development 



eommunity Volunteer 
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ECONOMICS INTERVIEWS 



*LAMAR GOULAND 



.TOM LANA^ 



GEORGE WEEDEN 



PAT JA*CKMAIT 



*BILL BARGER 



U.S^ Bureau of Labor Statistics 
Washington, DX. 



Economist 
Edonomls t ^ 
^ Economis 
Chief 



Dlvi^tfn of Living Conditions 
^nd ^ice Studies* ^ 

/ * . ' * ^ 
Division of. Living Conditions 
and Price. Studies 

■ y ) 

Division of Living Condition^ 
and Price Stu:4.ies 



C9nsumer Price I; 



,ch 



Bank of America 
* Los Angeles, California 'i> 



Associate Economist Econoqj^cs Department " 



Communication Workers of America 
w • ^ Englevaod, Colorado 

*WILLiAM ^HORNBHtS^* |Cepresentati,ve 

- Th^ Education Commission of the States 



Denver, Colorado 



PHILLIP VINCENT 
ROY FOUBES 



9^1±c Finance Econo- Department of Research and 
Vniist Information ^ 



Director 



*^ati6nal Assessment of Educa- 
tional Progress 



. / 



♦Interviewed by telephone 
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toDICAL INTERVIEWS 



. HAROLD :j. DDPUY 

"ARTHUR J. Mcdowell 

JEAN ROBERTS 
JACOB J. FELDMAN 

4 

* * O , 

t 

ROBERT E-^FUSCHBERG- 

" ► ' ^ 

ALICE E. HAYWOOD 

*GARRIE J. LOSEE 



National Center for Eealth S^tistics 
Hyattsvllie. Maryland- * ? 

Psychological Advisor division ofc Health Examina- 
tion Statistics 



Director 



Divisibn of Health Examina- 
tion Statistics 



Chief, Medical Statis- Division .of Health Examina- 



tics Branch 

Associate Director, 
for Analysis 

'Director4 



tion Staltistics 
Division of Analysis. 



Division O'f Health I^ntejrview 
Statistics 



Technical Information Office of the Center Director 
Officer 



Deputy Associate 
Director 



.Cooperative Health Statistics 
System 



SANDRA SURBER ^TH Chief, Scientific and Division of Operations 



ELIZABETH L. WHITE 



^ 'Technical Laformation 

Associate Director for Data Systems 
Data Systems 



IJANCY BROWN 



University of California 
• San Francis CO ♦ California 

Administrator, Healtfi 
Policy Program \ ^ * ^ 



*IR.VIN,J. LEHMANN^ 



Michigan Statgi University 
, 206 South Kedzie Hall 
East Lansing, Michigan 48824 ' 

Professor of Evaluation 
Services • 



^Intervifwed* by telephone 
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EDUCATION ' INTERVIEWS 



GEORGE KAPLAN 



Institute for Educational Leadership 
Washington^ D,C, . 

Coordinator, Washington 
'"Policy Seminar 



RICHARD k/hERLIG 



Chief State School Officers' Council 
^ Washington, D,C. - 

Dizjector, Dissemination 
Prcfject 



EDWARD ELLIS 
LARRY ,EUTCHINS 



REX HAGANS 



EDWARD TYLER 



National Instituflfe for- Education 
Portland, Oregon - , 

~^ ^ 

Chief, Constimer Information 

Branch 

^ Chief, School Practices and 
Service Divisibn ' ^ 

■ \ ■ ^ 

Northwest Regipnial Educational Laboratory 
' ■ Washington,' D,C* 

. , Program Director, Education 
and Work Program 

Specialist, Dissemination 
and Publication 



49 



OTHER INTERVIEWS^ 



CHESTS^R FIM ' 



Dirksen Senate Offrfce Building 
' ' Washington^ D.C, \ ' 
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Economic Programs . / . - 

CONSUMER SXPENDITTJRB SURVEY CCES) ^ / * 

Periodically, since 1888,. tke Bureau of Labor ^Statistics has conducted surveys 
of consj^r expenditures^ savings, - and income.* These surveys have been the 
dioly comprehensive souj^es of detailed information on expenditures, income, and 
changes in assets andiliafailitjies xelated to the socioeconomic and demographic 
chalracteristics of families in the Uiiited States'. • • 

The 1972-73 Consumer Expenditure Survey is the eightji major survey of this type, 
and the first since 1960-61. The 1972-73 survey consisted of two separate! 
surveys, 'each with a different data collection tecri^ique and sample: (1) a/ 
qtiaryerly panel survey in which each consumer unit in the sample was visited by 
an interviewer every three months over a iS-month period, and (l) a diary or 
record-keeping survey completed at home by the respondent family for two 1-week 
periqd^s. The survey was conducted over a 2-year period. The sample, of .23,000 
units was divided into two representative *subsamples of approximately 11,500. 
units for each stjrvey year. * . . 



CONSUMER PRICE INDEX • " . 

The Consumer Price Index is a monthly statistical measiure of th^ ave^rage change^ 
in prices of goods and services purchased by urban wage earners and clerical 
workers for day- to-day livings It is based on prices of about 400 "marketbasket" 
Items selected to represent all consumption goods and services purchased by 
these porkers. The quantity and quality of .these items is/kepc essentially 
unchanged between major revisions so that only price changes will b^measured. 
Prices are' collected in a Consumer Expenditure Suci/ey.from about 40,0d0 tenants 
and 18,000 retail establis^hments in 56 urban areas across the country. All 
^taxes directly associated with the purchase and use of the 400 items are in-, 
•eluded in the, index. * Since thetPI is based on the expenditures of a specific 
population group, it may not accurately reflect the eixperience^ of other families 
and- individuals with different buying habits. Though the CPI is often called 
the "Cost-of-Living Index," it measures only price change, which is just one of 
several important factors affecting liying costs* Area indexes do not measure 
differences in phe. level of prices among cities. They only. measure differences, 
in the level of prices among cities. They only measure the average change in 
prices for each^afea since th^ base period. - 
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Health Programs ' - 

MIIONAL HEAT^m SURVEY f^,/' • . ' • 

>The National Health Survey Act of 1956 provides for "the ektablishment and 
continuation of a National Health * Survey to ohtain information -about the ^ 
health status of the population in the United States, including the services 
received for or because of health conditions. .The responsibility for the 
development and conduct of that program is placed vith the National Centet 
for Health Statistics (NCHS), a researcfh-oriented statistical organization 
within the Health Services and Mental Health Administration (HSMHA) of the 
Department of Health, Education, and Welfare. Three separateSind distinct 
programs are- employed b^ NCHS in meeting the ^objectives of the Ac^~a house- 
hold health interview survey, a family of siirveys of ^ealth resources, and 
a heal^ ejsssi^ation survey, J 

TBS, HEALTH EXAMINATION SURVEt (HES) \ • 

The Health Examination Survey (HES) is one .of the three different programs 
employed by NCHS to accomplish the objectives of the National Health .Survey. 
It_collects data by drawing samples of Che civilian, noninstitutionalised 
population of the United States and, by means of medical and dental exam- 
inations and, varions^testd-^and measuremexfbs , uftder takes to characterize the 
population/ under study. This is the most accurate way to obtain definite 
diagnostic data on the preval^ce of certain medically defined illnesrses. 
It is the only way to obtain information on unrecognized and undiagnosed 
conditions— in some cases, even nonsymptomatic conditions, it is also the 
only way to obtain distributions o'f the population by a vari)ftty of physical, 
physiological, and psycl^ological measurements. 



The overall plan of the Health Examination Survey has been to conduct successive, 
separate programs in specific age ^segments'^of the ^civilian, nonxq$titutionalizfiiia 
U.S. population by means of medical and dental examinations ^ tests, and measure- 
ments. Thea,e successive programs, referred to as "cycles," have had a specific 
age segment for the target population and have -been concerned^ with certain 
specified' health aspects of that subpopiilatlon^ ' ^ 

During the period from 1959 - 1970 three separate survey programs were conducted: 

Cycle 'I Nov 1959 - December 1962 Ages 18-79 

The .examination was focused on certain chronic diseases » cardiovascular diseases, 
arthritis *and rheumati^m^ and diabetes. Also included were a dental examination; 
tests for ^visual and' auditory act*ity. X-rays, electroSardiographic tracings^^^^ - 
•blood chemistry^ tests, and numerous bociy measurements. The sample size of Cycle I 
was 7,710 persons, of which" 6,672 (86.5 percent) wete examined.' 

Cycle II , ' July 1963 - December 1965 Ages 6-11 

The examination was focxised primarily on various parameters of growth and devel- . 
optaent, but it also screened far heart disease, congeni.tal abnormalities, ENT 
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abaonoallt±ea» and neuromuaculoal^letal abnorpialltles. The size^of the sample 
was 7,417,. of which. 7,119 (96,0 percent) were examined/ 

Cycle m March- 1966 - March 1970 ^ Ages 12-17 

For Cycle II, the focus was on growth and development, A unique feature 5f the 
survey was that the same sample areas and housing 'units of Cycle II were used, 
again/ Thus, many of the Cycle II sample children were also examined in 
Cycle III, providing valuable longitudinal data. Of the total sample size of 
7,518 youths, 6,773 (9Q.1 percent) were examined. Of those examined, 2,271 
were examined in both cyc4.es. 

Cycle IV ' -r 19^1 - 1973 Ages 1-74 

« < 

following the Health Examination Purvey among youths in 1966-1970 this survey 
program also was requested to undertake the responsibility of ^nutrition sur- 
veillance. The first Health and Nutrition Examination Survey program started 
in 1971 was designed to measure and monitor nutrit;Lonal status of the population 
1-74 years and to collect other d^ta needed on health of tlie adult population, 
25 - 74 yfears. Measures 'of nutritional status include clinical evaluations, 
with special" att"ention to conditions indicative of nutritional problems, 
hematological and biochemcial determinations, and special skin, eye, and dental 
examinations. A subsample of the adults 2^ - 74 years of age also received 
a detailed examination designed to provide 3ata on current and unipet health" 
needs in this adult population* Persons were questioned ^about their health 
care needs as they see them and about the medical care they have sought and 
obtained for these needs. The examination was designed to provide informatipn 
on health problems found and sonie l|idications of health care needed. These 
data will' be interrelated to obtain assessments of how well the medical^»care 
system is working, • * 
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HEALTH INTERVIEW SURVEY * , * ' . * ^ 

The Seal th^ Interview Survey, one pf the programs conducted by the National Center 
for Health Statistics to gather Infonnatioh on the healifeff of the American peiple, 
has been in operation since July 1957. From information filtered in interviews, 
an attempt is m^ce.to describe the social demographic^ and economic as{)eyts oi 
illness ,^disabi£l,ty, and the use'^of medical service^. ' 

The health intervi ew cjuestionnaire co^^ists of a cQre of questions concerning 
^tems about whicxi' i^nformation has been collected each year. These basic^tems' ^ 
^include acute cqnditions a^d injuries, chronic conditions,- days of disability^ 
4ue to acute or chronic conditions, limitation of activity caused by^ chronic 
conditions or impaitment, hospitalization, and the social, economic, and 
demographic chara^eristics of the 'interviewed sample persons, ^u^ing recent 
years, medical care provided by physicians and dental cate have become core . 
items . ' 

Information on certain other health-related items has been collected in the sur- 
very periodically, usually at intervals from, two 'to five years. In the early 
years of the survey, these topics were described as rotating iteSis and consistad 
of measures such as mobility limitation due to chronic illness, dental care^ and . 
the proportion of hospitsd b'ills paid by insurance. As the survey developed, 
there was an -increasing demand for more detailed information about -some of the 
core items, such as detailed data on types of injuries, the duration of 
activity l^jnitation, the, accessibility of physicians' services, and-«conva-* 
lesenca following hospitalization. The need for this information led* to the 
expansioi^ of certain areas of the'basi^ questionnaire to provide for \he 
collection of these data at periodic intervals, introducing a slightljf differaat 
category of rotating items. ^ . ^ 

In addition, the questionnaire for a given year has usually included one or more 
special supplements. While most of the supplements, were originally planned for 
a 1-year collection period, some -of thei^^deal with topics for which trend 
information is needed. 'Since the interview is the ^most efficient method of 
collecting this type o^ data, certain items ^are added to the basic questionnaire 
on a planned schedule. These items, which might be described as rotating 
ssupplements include such topics as smoking habits, health insurance coverage, X-*ray 
exposure,, home care, the use of special aids, and personal health expenses. 
Other supplements, particularly those dealing with specific chronic conditions^ 
or impairments, have been added to the questionnaire on a l^ss regular basis. 
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Title: 



Area: 



How much impact do you think educational research has had on educational policy 
decisions made at the .following levels? 



Level • 



r • 



Qwsiderable. Impact 
^ Moderate Impact 
Some Impact . 
Very Little Impaqt 
^ No Impact 



Federal 


State 


2 . 


V' 


7 ' ' 


\ 4 


6 




4 ^ 


■ • V 



' Local 



No Response 



10 



How much impact do you'think the type of educational research you have been involved- in 
at National Assessment has on educational policy. decisions made at the following, levels? 



Level 



Considerab'le Impact 
Moderate Impact 
Some Intact 
Veiry Little Impact 
No Impact 



How mlicK utilization do yi 




FeS&Tal 



'6 



8 



State 



Local t 



. 1 



No Res't^onse 



10 



:he following^ groups make of educational research data? 
Utilization 



Very 



Teachers 

School Board Members 
Schqpl Administrators 
r Students 
Parents 

State Education Personnel 
Feder^^]. Educational 

Agency Personnel 
Educational Researchers 
College of Education 

Professors 
Statfe Legislators 
Federal Legislators 



Considerable 


Mo derate 


Some 


Little 


None 




3 


S 


10 


1 
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How much utilization, do you feel the<^foiroWing> groups mike of l^ational Assessment data? 



Utilization 



Teachers 

School Board* Members - * 

School Administrators 

Students r 

Parents ^ ' 

State Education Personnel 

Federal Educational 

Agency Personnel 
Educational Researchers 
College of Education 

Professors . 
State Legislators 
Federal* Legislator 



Cons i de r ab 1 e 


Moderate 


Some 


Very * 
Little 


None 


- 


• 


10 


8 


2" 
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_3 . 




3 


3 
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I4 
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1 


8 
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7 


6 r 
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.5 


' 10 
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1 ^ 


4 


9 


•'s 


1' 




1 - 


9 
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No 

Response 



Rank from one to eight the following types of research in terms- of jthe impact of fes^ 
on the general public, (i.e., the Amount of attention* paid by the public to the res< 
one for the most attention' to eight for^the least ; ' ' ^ 

^ ' 6 Agricultural research ^ . ' 

• 2 Aerospace research • 

Biological research ^"^^ ^' 

Economic research (e.g., mai^ting research) 
5,ducational Research ^ 
Industrial research (e^g.; product' development) 
Medical research <s 
Sociological researc^h . , ^ 



3 



Ifliich one of the following statements be^ describes how you feel ab<5ut the future 
impact of educational research? ' • • * 



11 



Educational res eardh will have increasing impact on educational 

. ' " ^ policy in the next fivfe years ' ^ 

• 7 Educational researchVill have about^ tfhe same ifj^act 'it jioes now^ 
oiilfeducatidnal policy in next* five years. • * 
' \ . ■ " ^ ' • . ' 

* , Educational research will have a decreasing "^i^npact^^dn--^ 

policy in the. next ^ve years. ^* . 

r^act°o£ .^EP^datf ^""'^"^ ;§t^ements best descr^Tjes^^fto^o^ou feel about i;he future 

7 . .16 . data will have an- increasing "impact 'on educational.tfblicys^in 

the next five years , / . * ' . 



V. - 



^ ^ N/yEP d&ta will hav.e'^ about, the same impact it 'does now on education^ 

policy in t^e ne^t £ive •years • ' ' ^ , ^ . 



In terms of recpgnit&iv of the,iiiipoftance of reseai^ch wif hin the profes$ioTis 
da you feel that edupatiolial researchers receive lesf^"about the same or more 
recpgnitidn xhai^^^e following types of researchers?* * ' \ 

^ . " ^ * / ^ ' - Amount o^f IJecognition , . " ^ 

* ^ ' — ~ — ' ^ ^ 2 

^ * \ Less 



About the- Sime 


More 


I .doji'^^lCndw 


4 




<• 


. , ■ 


' '1 ^ 




^3 . 


1 




" 4 - 


. * 6 


I 








5 


1 



1 



1 • 



Agricultural 'researchers ' 

Aerospace researchers > 

Biological researchers. . 

Economic reseai^chers ° ' ^ 

Industrial Research ersi« ' 

Jiedical cesea^chets^ • , ^ ^ 

, Sociological researchers 

^' /' * — . , — 

Name two conditions or. problems that you feel may* impede tjie utili^zation of educational 
research data by €he following groups :^ * - . - • ' ' ♦ - ' 



1'; 



1 5 



State and/or National legislators 
• 1- . ' fsfe pp.' 55-341 / 



4 



■4- 



Statcj, Departments of^^educatidi^ 

.1. 



Local* school 'district^ (personne^l, schqol board meinbers, othels) ' * V. 



*Nake two conditions* or problems that you* feel may impede the ut^-lization of 
educational research data by the' following groups: 

State and/or N^tioW Legislators ^ . , . * 

1. ^ Lack of; readable summaries of educational reseJtch*. \ 

'Complexity of information ii^ description. . ' 

^ Re^search data ;iot written in concise, clear form. ^ 

Researcl^^is often incomprehensible. v , - 

Too .bureaucratic — * political power groups /sinc'e taking ^research into - 
•consideration means reallocation .x>f* funds, . • ^ * ^ 

runds t^t ava^lahde for implementatjlon. v , ' 

Labk of pr(icedures ("what to do")*vhen problems are 'doted? * 

. ^Tumber of sta/f catsable of understanding results implications of research 

, Politiqar considerations are more important .than use of data in decision 
^making. '(Their aissumption may be' that^ money solves problems /'increases ^ 
v: quality, of education.) ^ -\ ' / _ * ' ^ 

, The need for mote definitive data. - *■ - • 

. Ihe long "production time" of gibbering research, data, wh^n .mosc policy^ " 

issues need to be resolved immediately, (or so ib is thought) . 

' / ' ■ - • V . 

, Illiteracy. / ' " . / ^ ' , ^ / 

Poor cdmmunicatidn- by researchers. ' - ^ . / , ' 

' \ * • 

^ Politics. * . * * ' * ■/ * - • 

/ Lack of -time, to give priority attention to education issues. ' • 
Lack -of commudication with educators., ^ - U 

Lack of time to deal' with the data properly or too bulky — not condensed 
to useable form.. Just tod much;' Too many studies doing s^me thing. Not 
coordinated. < » * ' . ' 



NOT5:/'-*'Wro.no responses. ' 



• y'^Name^tt^i' conditions or problems that you^f^el impede the utilization of 
^ educational research d^ta the foildwing groups:/' 



I 



State -and/qr National^ Legislators ^ ^ ^ 

2. 'Lack of w^l defined^ general theory. . * * 
i Conflicting informatiod / J 



* ) ?oor' contact w.ith legislative ^taffs. -'^^ 

^ - . ^ ^. Contradictoi^Nand not prescriptive results. 

' ' ; ' ^ P^ogrammatiCj implications vfrom »res^rch unclear. 

' ^ i-ack of overall^ perspective of a*^ educational resel^rch design. 

. ' ^ Slowness of data. v t 

Data irom sufficient studies" do not cTeaoa an integrated picture of the 
* stsflle of education or. problem^^ thut need attention. 

^ * * 

- The .need to make 4ata understandable such that conclusions may b>'e drawn 
froni.it. . • ' ■ , 

- , . -'^ <^ ■ . • • • . ^. - • 

*» . ^ Obsecufe and irreleyant reports. : ^ 
^ • \ j/ ' " 

. ' \ I*ack of understanding by legls^lators ("xruat". - "credibility")* 

- - ' " <^ * ^ ' - 

- ,The unwillingness to take time to read the research. ^ ; 

. ^ Mistrust of educators. * ' . . ^ • •, L , 

Lack of interest; i.e'/, -education is not in the pu&lic eya a politically 
^ * ' "hot" issue 'to concentrate on.i ' ' 



"hot"N issue 'to concentrate on. 
.The wapning guality of feducation research. 



OS 





pXE: Four.no responses. , ^ - 



Name two condlti'iMis or problems- that you feel^^ay impede the- .utilization of 
educational Research data by 'the followihg groups.: *\ ' 

ttate Departments of Education / ' [ \ ' . ' 

. "nie h§ed for more d^firiitiv^ data, * ' • * . , . " 

Lack of readable s^^^ries of e^iucatioiial research. ' • ' . 

Comple^rty of informatibn* in description. 3 ' • 

\ / ' ^ ^' . ^' . h ' ' " ^ - ^ ^ \ ^ 

L^ck o^ time and- funds to jnake practical* usef' of research data. 
Research -not conclusive — can divide into •factioni. ' ' ^ 



as. I 



For, most gart £00 bureaucratic 

Lack',df* procedures ("what to do^) whp' problems are ^noted? " * 

^ Niimber (>f staff capable of unde^^ding r'esults impli|cJfe:ons of research^t 

La| timS between current fads and Research*' . *' v ^ * 

' ' • * '* ^ ' ; , 

' -Budget problems. \ ' ' ' 

.Lack 'of funds to' ^deqa^tely im^eiJtei^t new e'duc;ation .programs* * • 

' \ " ' ' ' ' ' ' 

Politics, lack of staff expertise, money^ . . » 

Poor communication systems to LEA's. ^ ^ . ' . ' . 

Budget problems; 

Growth of doUectiVe' bargainingWy well "back out" certain e4ucational ^ 
'.options ti(at "research may suggest are not^ effective ^(e'.g- , . teaqher/pupil 
ra,tio limits are "bargained" increasingly. , Even If ^research concluded ^ 
-that ^^ctive teaching cojuia take^ place within rat'i®s as high'as 'USO/ 
I bargaining agreements would. preclude • siich^n' option) . \ ' » 



0 ■ 



A 



\ 



riOT^: ^ Thre^ lio responses , 
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Name -two conditions or problems that you feel may impede the utilizatiy of 
educational research data by the following groi^ps: 



'State Departments' of Education^ 



/ 



2* .The need to ittake data understandable such that conclusions may be drawn 
from it. ' ^^ 

♦ * « 
Lack of sophistication noitded to understand research. 



Conflicting inforlnation.*^ 



Data not.jLn^focm tl^^t-^an be practically utilized. by^ state" departments. 



Not we^' enough organized tko way of getting hold of all dat^. 

Too unilavolved ;in actual education. • y ' ' \ " • 

Funds. There are nore pressing sociil issue^ to be dealt wxtli unrelated 
to or^ unaffected by existing research. ' . * 



Laziness-. , 

l^ck o^ 'staff to vimplement program' *<Shanges las avre^ult of resfearch. 

• " ' ' " * • » •' ^' ' 

- ' ' -DlssemiQation prqblei6s. ^ ^ ' V ' 

Budget. Each department ^vithin state department* doing thexr.own thing. 
* Qpofdinationi' ^ ^ •/ ^ . ^ - , , 

. /'/Lac:k of specialist \)t trained personnel to. be -the caordipator Ijetween 
N state 'and local school- distripts. ' ' : ' 

. • . . ' / ' • > ■ ; 

Emotional turmo?.l caused by. some research And testing programs. . 
Weak position of 'resparph gryup within state, depkrtnaents. . . * 



■ ♦ 



NOTE: Five no responses. 



'Naine two conditions or problems thatf you feel may' ippede the utilization of 
educati9nal research data by the following groups: 

Local School Districtv (Personnel, Serhool Board -Members, Others) 

\ ' « • ' 

1^ ' The need for^more definitive data. \ * ' * 

Research' needs tg^be ic^terpreted ''f(5¥'-'^assroom implicatioi^*. 
Lack of good rea(iiable; summaries of educational research. 
Cqmplexity-iof *in£bi3natioa''in desctfiption.* ' ^ 

Interpretation of research' cjat^a ia form' that can be utilized in classroom, i 

• . ' ' \' .• ■ : • v\' 

->^tudies, den' t giv6 clear mandate for action ° , • 
^ ' ' > . ■ ' ' ' '■ • ■ 

♦ s . A. natural district, of "s tali's tic^^' (too far removed from "real" world.). >• 

, Many local' school districts-'^canhot afford dunlicatiDn of, i.e., NAEP model. 

/ Funds not s^vailable- for implementation. u * • * 

. "Lfick of procedures ("what to do "3 whea problems' are noted'. * ' 

Num6^r*of staff capable .of understanding results^iihplications of research. 

- ' • • ^ • / ' ' -^'^ ^ 

' LWal*^cisions need to be, Based^on" spnfe .type of local ''cfata 



1 



% A3 wifh, mo^' .behavioral resea3fch,^-vthfr^lack of ".unequivocal findings is often; 
more' can^usi^g ^ thfen he:j.pful.. to iCtie practitioner. ( - ' . ' *^ ^ ^ 

Ignorance of ^iterati^e. ^ - • - , ^» ' > \ . 

r ' ^ , ' ^^''v # / ^ '\ ^'^ ' . \ I ^ . ' 

Lack of funds.! ^^Doesti^t apply to. our 4oc*al district ^ we're okay> Too " 

• many, often .pressing schoo'L p.roblemfe. r> 

Lack >of iocal uti3^.ty of .i^ost: ek;icatjtqn rese^irch. * , ' * ^ 

* Local scliool peopre njaed^ training .or* leadership for* utilizing . this data' 

*• y into lise'aj)le prigrams i£^schoblg^ ^ * * - 

* . ' . . , y 1 ' •> 

'Re§ourc^s (mon^y, $taf5 exper^tkse, *lack ofc)\ ^ . * ' * * 

^ r ' I" ' ^ ^ ' ' " * ; ' 

Lack of definitive stfudies. — "ajiswers." ^ ' . • f , ^ . 



Name Wo ^conditions or problems that you feel may impede the utilization of 
e^ucatioiJal research' data by the following groups : ' ; ^ 

Local School District (Personnel, School Board Members, Others) 

2. Jhe need to^make data understandable such that conclusions may l?e .drawn* 
from it. ' ' • ^ ' ' * - • 

• Classroom^ implication^* . * • ' * ^ ^ , 

\ ■ ^ • ■ ■ 

Lack of sophistication needed to understand research reports. 
^ Conflicting information. . * - , 

Conflicting research on same problem* 
Results often contradictory.^ 

Anvinability to understand 'the foundations -of t!ie research. 

^ , Locals perceive national education research, doasn't reJLate \to their 
^ problems^' ' . ^ , • . . 

'Competing aspects of social' pol^^cieJ at local level% e,^g., there are mpre 
pressing^ social issues, to be dealt with, unrelated or unaf f ected ,by exist- 
^ /ihg i^search. ' ' • . * 

. Itisularity — , "t"h^.s doesn't apply to our unique, .etc." 

• Lack, of -staff to> implement program changes as ^ result of research. 

. " ^ * ' ^ . , , ^ J ^ ' , . • 

Lack of scientific training by ^thesd people msijkesi it unlikely t;hat they 
^ca^undersfand^ specific <esearph'*restilps , ^Whereak xhey could understand 
and U3<2enerali2atfons ^b<^ut cause and effect if researchers could <iq 
a^equateXtudies tc 'ge^ them tHis information. ^ 

Obsecure ai\d^>drrelevant reports. — \ * . 

In small distri(^s; ^the lack of qualified personnel yCo. work witlf the data. 

Ne^^^iuxxding to \carry through these" programs . ; 

/.Lack of relevance 4lid benefit to individual systems; not meeting; the sp«-^ 
cific rleeds of individual systems. 

\ v^F^t^' of numbers, statistics. ► ^ 
NOTE: - Two no' responses . ' * ' " •, .. ' ■ 



